Skip to content

feat(mypyc): Enable incremental compilation#7558

Open
VaggelisD wants to merge 3 commits intomainfrom
mypyc-separate-flag
Open

feat(mypyc): Enable incremental compilation#7558
VaggelisD wants to merge 3 commits intomainfrom
mypyc-separate-flag

Conversation

@VaggelisD
Copy link
Copy Markdown
Collaborator

@VaggelisD VaggelisD commented Apr 24, 2026

What

Switch sqlglotc's mypyc build to separate=True, plus the two small sqlglot-side tweaks needed to make it work end-to-end.

The separate=True flag gives each compiled module its own shared lib + shim, so mypyc only has to regenerate / recompile the modules whose source actually changed. Clean builds are roughly the same wall-clock; incremental rebuilds after a one-line edit drop dramatically once the cache is warm.

Three small changes, all behind the [c] compile path:

  • sqlglotc/setup.py: pass separate=True to mypycify(). Everything else in the build recipe is unchanged.
  • sqlglot/__init__.py: the existing *__mypyc*.so bootstrap preloader was written for the old monolithic build where a single hash-named .so sat at the package root. Under separate=True the per-module shared libs (e.g. errors__mypyc.so) live next to their .py siblings and resolve through Python's normal import machinery via the shim. Skip preloading those; keep the legacy behavior only for .so files that don't have a .py sibling.
  • sqlglot/optimizer/__init__.py: swap the eager top-level re-exports for PEP 562 lazy __getattr__. Under separate=True's cross-group init ordering, the previous eager from sqlglot.optimizer.optimizer import ... could fire while sqlglot's own __init__.py was still mid-bootstrap, hitting a circular-import ImportError on from sqlglot import Schema, exp. Lazy imports defer the cycle to first use.

@VaggelisD VaggelisD changed the title feat(mypyc): enable separate=True for faster incremental compilation [CLAUDE] feat(mypyc): Enable incremental compilation Apr 24, 2026
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch 2 times, most recently from 8db1da4 to e8129f7 Compare April 24, 2026 13:47
Comment thread sqlglot/optimizer/__init__.py Outdated
Comment thread sqlglot/optimizer/__init__.py
Comment thread sqlglot/optimizer/__init__.py Outdated
Comment thread sqlglot/__init__.py Outdated
VaggelisD added a commit that referenced this pull request Apr 27, 2026
…UDE]

Review feedback from #7558:

- sqlglot/__init__.py: drop the *__mypyc*.so bootstrap preloader; legacy
  hash-named .so layout doesn't happen with separate=True.
- sqlglot/optimizer/__init__.py: rename _t to t, tighten the docstring.

Drop py3.9 support for sqlglot[c]:

sqlglot-mypy 1.20+ doesn't ship a py3.9 wheel and the older 1.19.x line
(which has a universal py3-none-any wheel) predates the separate=True
fixes, so py3.9 CI was crashing with "vtable not computed yet" while
trying to build sqlglotc.

  - setup.py: gate the [c] / [rs] extras on python_version >= 3.10, so
    `pip install sqlglot[c]` on 3.9 is a no-op (you get pure-Python sqlglot).
  - sqlglotc/pyproject.toml: bump requires-python to 3.10, pin
    sqlglot-mypy >= 1.20.0 in build + dev requires.
  - Makefile: install-devc / install-devc-release skip the sqlglotc
    build on 3.9 instead of failing.

sqlglot itself still supports py3.9; only the optional compiled extension
requires 3.10+.
…[CLAUDE]

sqlglot-mypy 1.20.0.post3 makes mypyc's separate=True compilation work
correctly on real-world projects (cross-group class inheritance, generator
helper classes, non-ext subclasses with fast methods, mutually-dependent
compiled modules). Turning it on for sqlglotc cuts incremental rebuild
time dramatically — a 1-file edit goes from ~110s (full monolithic rebuild)
to ~3s.

Three small changes:

- sqlglotc/setup.py: pass separate=True to mypycify(). Produces one
  shared lib + shim pair per module instead of one monolithic shared lib;
  mypyc only regenerates the changed module's C on rebuild.

- sqlglot/__init__.py: tighten the `*__mypyc.so` bootstrap preloader.
  Under separate=True, per-module shared libs (e.g. errors__mypyc.so)
  live next to their .py sources and are resolved via normal dotted
  imports through the shim. Skip preloading those; keep the legacy
  preload behavior only for monolithic-hash .so files that lack a .py
  sibling.

- sqlglot/optimizer/__init__.py: make the top-level re-exports lazy via
  PEP 562 __getattr__. Under separate=True's eager cross-group init, the
  previous eager `from sqlglot.optimizer.optimizer import ...` could
  trigger while sqlglot's own __init__.py was mid-flight, hitting a
  circular-import ImportError on `from sqlglot import Schema, exp`.
  Lazy imports defer the cycle to first use.
VaggelisD added a commit that referenced this pull request Apr 27, 2026
…UDE]

Review feedback from #7558:

- sqlglot/__init__.py: drop the *__mypyc*.so bootstrap preloader; legacy
  hash-named .so layout doesn't happen with separate=True.
- sqlglot/optimizer/__init__.py: rename _t to t, tighten the docstring.

Drop py3.9 support for sqlglot[c]:

sqlglot-mypy 1.20+ doesn't ship a py3.9 wheel and the older 1.19.x line
(which has a universal py3-none-any wheel) predates the separate=True
fixes, so py3.9 CI was crashing with "vtable not computed yet" while
trying to build sqlglotc.

  - setup.py: gate the [c] / [rs] extras on python_version >= 3.10, so
    `pip install sqlglot[c]` on 3.9 is a no-op (you get pure-Python sqlglot).
  - sqlglotc/pyproject.toml: bump requires-python to 3.10, pin
    sqlglot-mypy >= 1.20.0 in build + dev requires.
  - Makefile: install-devc / install-devc-release skip the sqlglotc
    build on 3.9 instead of failing.

sqlglot itself still supports py3.9; only the optional compiled extension
requires 3.10+.
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch from 6ac5ab9 to 949c883 Compare April 27, 2026 17:21
…UDE]

Review feedback from #7558:

- sqlglot/__init__.py: drop the *__mypyc*.so bootstrap preloader; legacy
  hash-named .so layout doesn't happen with separate=True.
- sqlglot/optimizer/__init__.py: rename _t to t, tighten the docstring.

Drop py3.9 support for sqlglot[c]:

sqlglot-mypy 1.20+ doesn't ship a py3.9 wheel and the older 1.19.x line
(which has a universal py3-none-any wheel) predates the separate=True
fixes, so py3.9 CI was crashing with "vtable not computed yet" while
trying to build sqlglotc.

  - setup.py: gate the [c] / [rs] extras on python_version >= 3.10, so
    `pip install sqlglot[c]` on 3.9 is a no-op (you get pure-Python sqlglot).
  - sqlglotc/pyproject.toml: bump requires-python to 3.10, pin
    sqlglot-mypy >= 1.20.0 in build + dev requires.
  - Makefile: install-devc / install-devc-release skip the sqlglotc
    build on 3.9 instead of failing.

sqlglot itself still supports py3.9; only the optional compiled extension
requires 3.10+.
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch from 949c883 to 176b79a Compare April 28, 2026 07:14
VaggelisD added a commit that referenced this pull request Apr 28, 2026
…fy [CLAUDE]

Temporary debug aid for #7558: integration tests are producing a
~530KB wheel that's missing all shared-lib .so files, but local builds
produce the full 18MB wheel. Add stderr prints so pip's failure output
shows what python/mypyc/source-dir setup.py actually saw, plus a hard
exit if mypycify returns zero shared-lib Extensions, so we get a clear
error instead of a silently-broken wheel.

Will revert once we've identified the root cause.
VaggelisD added a commit that referenced this pull request Apr 28, 2026
…fy [CLAUDE]

Temporary debug aid for #7558: integration tests are producing a
~530KB wheel that's missing all shared-lib .so files, but local builds
produce the full 18MB wheel. Add stderr prints so pip's failure output
shows what python/mypyc/source-dir setup.py actually saw, plus a hard
exit if mypycify returns zero shared-lib Extensions, so we get a clear
error instead of a silently-broken wheel.

Will revert once we've identified the root cause.
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch from ff7a90c to 69aed12 Compare April 28, 2026 08:38
VaggelisD added a commit that referenced this pull request Apr 28, 2026
…fy [CLAUDE]

Temporary debug aid for #7558: integration tests are producing a
~530KB wheel that's missing all shared-lib .so files, but local builds
produce the full 18MB wheel. Add stderr prints so pip's failure output
shows what python/mypyc/source-dir setup.py actually saw, plus a hard
exit if mypycify returns zero shared-lib Extensions, so we get a clear
error instead of a silently-broken wheel.

Will revert once we've identified the root cause.
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch from 69aed12 to 041b5ff Compare April 28, 2026 08:48
VaggelisD added a commit that referenced this pull request Apr 28, 2026
…fy [CLAUDE]

Temporary debug aid for #7558: integration tests are producing a
~530KB wheel that's missing all shared-lib .so files, but local builds
produce the full 18MB wheel. Add stderr prints so pip's failure output
shows what python/mypyc/source-dir setup.py actually saw, plus a hard
exit if mypycify returns zero shared-lib Extensions, so we get a clear
error instead of a silently-broken wheel.

Will revert once we've identified the root cause.
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch from 041b5ff to 515d038 Compare April 28, 2026 08:59
…fy [CLAUDE]

Temporary debug aid for #7558: integration tests are producing a
~530KB wheel that's missing all shared-lib .so files, but local builds
produce the full 18MB wheel. Add stderr prints so pip's failure output
shows what python/mypyc/source-dir setup.py actually saw, plus a hard
exit if mypycify returns zero shared-lib Extensions, so we get a clear
error instead of a silently-broken wheel.

Will revert once we've identified the root cause.
@VaggelisD VaggelisD force-pushed the mypyc-separate-flag branch from 515d038 to ab9622e Compare April 28, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants